Depth map super-resolution processing method

ABSTRACT

The present invention discloses a depth map super-resolution processing method, including: firstly, respectively acquiring a first original image (S 1 ) and a second original image (S 2 ) and a low resolution depth map (d) of the first original image (S 1 ); secondly, 1) dividing the low resolution depth map (d) into multiple depth image blocks; 2) respectively performing the following processing on the depth image blocks obtained in step  1 );  21 ) performing super-resolution processing on a current block with multiple super-resolution processing methods, to obtain multiple high resolution depth image blocks;  22 ) obtaining new synthesized image blocks by using an image synthesis technology;  23 ) upon matching and judgment, determining an ultimate high resolution depth image block; and 3) integrating the high resolution depth image blocks of the depth image blocks into one image according to positions of the depth image blocks in the low resolution depth map (d). Through the depth map super-resolution processing method of the present invention, depth information of the obtained high resolution depth maps is more accurate.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to the field of computer image processing,and in particular, to a depth map super-resolution processing methodbased on image matching

Related Arts

A super-resolution processing technology is one of the research hotspotsin current disciplinary fields such as computer vision and image videoprocessing, for processing natural images with low resolution and lessdetailed information and generating a high resolution image containingmore detailed information, which is a technology that improves theoriginal image resolution. The super-resolution processing technologyhas been widely used in fields such as high definition movie andtelevision, image compression, medical imaging, video surveillance andsatellite image analysis. Especially in recent 30 years, thesuper-resolution technology has been widely and deeply studied. A depthmap contains three-dimensional depth information of an object in ascene, and plays an important role in constructing a three-dimensionalvisual scene. A good high resolution depth map can project acorresponding color image into a three-dimensional scene to display aclear and complete effect, which provides a powerful support forconstructing an efficient and high-quality three-dimensional scene.Therefore, acquiring a high-quality high resolution depth map is ofgreat importance in three-dimensional vision.

In the existing methods of acquiring a depth map, the depth map isacquired with a laser depth scanning method, the method can acquire ahigh-quality high resolution depth map, but the acquisition method hashigher requirements for the equipment and technology, resulting in thatthe cost is high, and only one point is scanned once, the acquisitionspeed is slow, and it is difficult to meet a real-time requirement.Also, a scene is directly shot and acquired through a depth camera, forexample, a time-of-flight (TOF) camera or the like, to rapidly obtain adepth image in real time; however, the method can only obtain a lowresolution depth map, and further processing is required for obtaining ahigh resolution depth map. In the existing processing methods,super-resolution processing is directly performed on the depth map witha super-resolution method, but the quality of the high resolution depthmap in actual scene rendering cannot be obtained after processing, whichis of little practical importance.

SUMMARY OF THE INVENTION

The technical problem to be solved in the present invention is toprovide a depth map super-resolution processing method with which depthinformation of a high resolution depth map processed is more accurate,so as to make up for the shortcomings of the prior art.

The technical problem of the present invention is solved through thefollowing technical solution:

A depth map super-resolution processing method, including: firstly,performing image acquisition on the same scene in a first position and asecond position, and respectively acquiring a first original image (S1)and a second original image (S2); acquiring a low resolution depth map(d) of the first original image (S1); secondly, performing the followingprocessing: 1) dividing the low resolution depth map (d) into multipledepth image blocks; 2) respectively performing the following processingon the depth image blocks obtained in step 1); 21) performingsuper-resolution processing on a current block with multiplesuper-resolution processing methods, to obtain multiple initial highresolution depth image blocks having the resolution the same as that ofthe first original image (S1); 22) traversing the multiple highresolution depth image blocks obtained in step 21), respectivelycombining corresponding image blocks in the first original image (S1)which correspond to the current block, and synthesizing multiple imageblocks corresponding to the second original image (S2) by using an imagesynthesis technology according to a relative position relationshipbetween the first position and the second position, which are defined asmultiple synthesized image blocks; 23) traversing the multiplesynthesized image blocks obtained in step 22), respectively calculatinga matching degree between each synthesized image block and acorresponding block in the second original image (S2) which correspondsto the current block, determining the synthesized image block with thehighest matching degree, and determining a high resolution depth imageblock corresponding to the synthesized image block with the highestmatching degree as an ultimate high resolution depth image block of thecurrent block; and 3) integrating the high resolution depth image blocksof the depth image blocks into one image according to positions of thedepth image blocks in the low resolution depth map (d), to obtain asuper-resolution processing map of the low resolution depth map (d).

Compared with the prior art, the present invention has the followingbeneficial effects:

The depth map super-resolution processing method of the presentinvention uses multiple existing super-resolution methods torespectively perform super-resolution processing on each block of adepth map, generates synthesized image blocks corresponding to a secondoriginal image according to generated high resolution depth map resultsand by respectively combining corresponding first original image blocks,matches the generated synthesized image blocks with the known secondoriginal image block, and obtains a high resolution depth image block tobe found through the synthesized image block with the highest matchingdegree. In the method, a high resolution depth map is determined basedon a matching degree between a synthesized image and an actual image,and the determined high resolution depth map more matches and is closerto the actual situation, that is, depth information of the highresolution depth map is more accurate, so that the processed highresolution depth map is of more practical significance and use value.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow chart of a depth map super-resolution processing methodaccording to a specific embodiment of the present invention; and

FIG. 2 is a schematic diagram of the principle of projection andrestored imaging in a depth map super-resolution processing methodaccording to a specific embodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

The present invention is described in further detail below withreference to embodiments and the accompanying drawings.

The present invention is conceived as follows: by study on asuper-resolution technology and a Depth-Image-Based-Rendering (DIBR)technology, the quality of a high resolution depth map restored by thesuper-resolution technology is reversely verified by using a matchingresult of a synthesized image blocks and an original image block. In thespecific embodiment, block processing is first performed on an image, adepth image block is restored to a resolution level the same as that thecorresponding color image through multiple existing super-resolutiontechnologies and then is projected to a three-dimensional space by usingthe restored depth image block and the corresponding image blockinformation, then new synthesized image blocks are obtained through avirtual camera and a three-dimensional scene, the synthesized imageblocks are matched with a collected initial image, and a desired highresolution depth image block corresponding thereto is found through thesynthesized image block with the best matching effect. Each block in thelow resolution depth map is processed as above, a high resolution depthimage block of each block is obtained, and finally, the high resolutiondepth image blocks are integrated to obtain a high resolution depth mapafter super-resolution processing.

In the specific embodiment, a depth map super-resolution processingmethod is provided, and super-resolution processing is performed on alow resolution depth map of a first original image S1. Respectively intwo different positions, image acquisition is performed on the samescene in a first position and a second position, that is, a firstoriginal image S1 and a second original image S2 are acquiredrespectively, and then a low resolution depth map d of the firstoriginal image S1 is acquired. When the low resolution depth map d isacquired, a depth camera, for example (but not limited to), atime-of-flight (TOF) camera, can be used, image acquisition is performedon the scene in the first position, and the low resolution depth map dof the first original image S1 is directly acquired. After the processedobject is obtained, the process proceeds to the processing steps asshown in FIG. 1:

P1) The low resolution depth map d is divided into multiple depth imageblocks. In this step, considering that different regions of the depthmap have different features (such as gradient condition), thesuper-resolution method most suitable for each region may also vary, andthus block processing is performed on the depth map, to respectivelyfind a super-resolution processing method most suitable for each block.A block processing method has many implementation manners, which are allapplicable to this specific embodiment and are not specificallydescribed herein.

P2) The following processing is performed on the depth image blocks:

P21) super-resolution processing is performed on a current block withmultiple super-resolution processing methods, to obtain multiple initialhigh resolution depth image blocks having the resolution the same asthat of the first original image S1.

In this step, the multiple existing super-resolution processing methodsinclude bi-cubic interpolation, new edge direction interpolation, a Kneighborhood embedding method, a sparse presentation method and thelike, and the processing methods have their characteristics and all canbe applied to this. For example, super-resolution processing isperformed on a current depth image block respectively with r existingsuper-resolution methods, which is processed into a high resolutionimage with the resolution the same as that of the first original imageS1, to obtain r corresponding high resolution depth image blocks. Theobtained multiple high resolution depth image blocks are defined as aset ID, and ΔD is set as any one high resolution depth image blocktherein.

P22) New synthesized image blocks are obtained by using an imagesynthesis technology, which is specifically: traversing the set ID,respectively combining corresponding image blocks in the first originalimage S1 which correspond to the current block, and synthesizingmultiple image blocks corresponding to the second original image S2 byusing an image synthesis technology according to a relative positionrelationship between the first position and the second position, whichare defined as multiple synthesized image blocks and set as a set IS.

In this step, when the image synthesis technology is adopted, thecorresponding image blocks are first projected into a three-dimensionalspace, and then new image blocks are generated based on athree-dimensional scene. The step specifically includes the followingsteps: a) projecting the corresponding image blocks in the firstoriginal image (S1) into a three-dimensional space by using a referencecamera with a depth-image-based-rendering method, that is, DIBR method,according to depth information of the high resolution depth imageblocks; and b) setting the center of a virtual camera by making thecenter of the reference camera correspond to the first position andaccording to the relative position relationship of the second positionrelative to the first position, and imaging a scene of thethree-dimensional space obtained in step a) into a two-dimensional planeby using the virtual camera, so as to obtain a synthesized image block.In a process of restoring the two-dimensional plane, relative positionsof the reference camera and the virtual camera are set according to therelative position relationship between the second position and the firstposition. The second position corresponds to the second original image,and thus the synthesized image block is an image block corresponding tothe second original image.

As shown in FIG. 2, it is a schematic diagram of the principle ofprojection and restored imaging during image synthesis. The center ofthe reference camera is located at the point O, the virtual camera islocated at the point O1, and the relative position relationship of thepoint O1 relative to the point O is equivalent to the relative positionrelationship of the second position relative to the first position. Asshown by the arrow A, it is a schematic diagram of projection into athree-dimensional space; the reference camera projects a pixel point p₁in an image block of the first original image S1 into athree-dimensional space, which corresponds to Pw. As shown by the arrowB, it is a schematic diagram of restoration and imaging into atwo-dimensional plane; the virtual camera restores the point Pw in thethree-dimensional space into the two-dimensional plane, whichcorresponds to the pixel point p₂.

In this embodiment, specifically, when projection is conducted in stepa), the corresponding image block is projected into thethree-dimensional space according to the following equation:(X _(W) ,Y _(W) ,Z _(W))^(T) =K ₁ ⁻¹ d ₁ p ₁wherein the center of the reference camera is the center of the worldcoordinate system, that is, the coordinate of the center of thereference camera is (0,0,0)^(T), and a direction observed from thereference camera is a z-axis direction of the coordinate system. p₁indicates position information of a pixel point p₁ in the correspondingimage block in the first original image S1, in a homogeneous form, thatis, the value of the third dimension is 1. For example, the position ofthe pixel point p₁ in the first original image S1 is (x1, y1), and pi inthe equation is (x1, y1, 1). d₁ is depth information of a correspondingpixel point p₁ in the high resolution depth image block, K₁ is abuilt-in parameter matrix of the reference camera, and(X_(W),Y_(W),Z_(W)) is a coordinate of a point in the three-dimensionalspace into which the pixel point p₁ is projected, which, as shown inFIG. 2, is the coordinate of the point Pw in the three-dimensionalscene. Certainly, there are many specific manners in the DIBR method toachieve projection from a two-dimensional image to a three-dimensionalscene, the above equation is merely one of those listed, and otherprojection manners are also applicable to step a).

Specifically, when the scene of the three-dimensional space is imaged tothe two-dimensional plane in step b), the scene of the three-dimensionalspace is imaged to the two-dimensional plane according to the followingequation:d ₂ p ₂ =K ₂ R ₂ P _(W) −K ₂ R ₂ C ₂

wherein the center of the reference camera is the center of the worldcoordinate system, that is, the coordinate of the center of thereference camera is (0,0,0)^(T), and a direction observed from thereference camera is a z-axis direction of the coordinate system. C₂ is acoordinate of the center of the virtual camera, R₂ is a rotation matrixof the virtual camera, K₂ is a built-in parameter matrix of the virtualcamera, P_(W) is a coordinate of a point in the three-dimensional spaceobtained in step a), p₂ and d₂ are position and depth information of thecorresponding pixel point in the synthesized image block obtainedthrough imaging to the two-dimensional plane; an operation result on theright of the equation is converted to a homogeneous form: m(x, y, 1), p₂is (x, y), and d₂ is the coefficient m. In the above equation,acquisition of the relative position relationship of the second positionof the second original image S2 relative to the first position affectsspecific values of the parameters C₂ and R₂, and an image block finallyrestored and imaged corresponds to the second original image S2.Similarly, there are many implementation manners of restoring atwo-dimensional image from a three-dimensional scene, the above equationis merely one of those listed, and other restoration manners are alsoapplicable to step b).

Through the image synthesis in step P22), corresponding new synthesizedimage blocks are synthesized for the high resolution image blocks, whichform a set IS.

P23) Upon matching and judgment, an ultimate high resolution depth imageblock is determined, which is specifically: traversing the set IS,respectively calculating a matching degree between each synthesizedimage block and a corresponding block in the second original image S2which corresponds to the current block, determining the synthesizedimage block with the highest matching degree, and determining a highresolution depth image block corresponding to the synthesized imageblock with the highest matching degree as an ultimate high resolutiondepth image block of the current block.

In this step, which high resolution depth image block in the multiplehigh resolution depth image blocks obtained in step P1) is asuper-resolution processing result closest to the actual situation isjudged based on the matching results between the new synthesized imageblocks and the original image block. The synthesized image block ΔS mostmatches the original image block, and the high resolution depth imageblock corresponding to the synthesized image block ΔS is thesuper-resolution processing result closest to the actual situation, soas to determine a high resolution depth image block aftersuper-resolution processing closest to the actual situation. The methodof calculating an image matching degree may be (but not limited to) theMinimum Mean Square Error matching method.

Through step P22) and step P23), the first original image S1 isprojected into the three-dimensional space through the generated depthimage information with the same resolution by using the DIBR method,then new synthesized image blocks are obtained by using thethree-dimensional scene to be matched with the acquired second originalimage, and the matching result is used as prior knowledge for the depthmap super-resolution, so as to obtain a high resolution depth imagewhich is reasonable and valuable.

P3) The high resolution depth image blocks of the depth image blocks areintegrated into one image according to positions of the depth imageblocks in the low resolution depth map d, to obtain a super-resolutionprocessing map of the low resolution depth map d.

In this step, the high resolution depth image blocks obtained in stepP2) are integrated into a complete image, to obtain a high resolutiondepth map of the low resolution depth map d. Preferably, after theintegrating, the method further includes smoothing the complete highresolution depth image. The smoothing is performed mainly inconsideration of an image overlapping region. In case of smoothing, itis feasible to use (but not limited to) a common mean method.

According to the depth map super-resolution processing method in thespecific embodiment, through the above steps, a processing result, i.e.,a high resolution depth map, is finally obtained. In the method, basingon the matching result between the new synthesized image block and theoriginal image block is the precondition of positively correlating withthe matching result between the high resolution depth map obtainedthrough processing and the high resolution depth map of the actualsituation, so as to determine which image block in the multiple highresolution depth image blocks obtained with multiple super-resolutionprocessing methods is most accurate and closest to the actual situation.That is, with the processing method in the specific embodiment, relativeto the existing method of direct super-resolution processing on a lowresolution depth map, the high resolution depth map obtained is muchcloser to the actual situation, depth information of the high resolutiondepth map is more accurate, is of more practical significance andvaluable. In addition, processing each block in the low resolution depthmap respectively by using advantages and characteristics of differentsuper-resolution methods fully ensures that depth image blocks havingdifferent features can obtain super-resolution processing methods mostsuitable for their image characteristics, and makes sure that multipleprocessing results include a processing result closest to the actualsituation. The processing method in the specific embodiment gives fullplay to characteristics and advantages of multiple super-resolutionmethods, well fuses advantages of the existing super-resolution methodsinto super-resolution processing of the depth image, and can restore ahigh resolution depth map which is of practical significance andvaluable.

The above contents are further detailed descriptions about the presentinvention in combination with specific preferred embodiments, but itcannot be concluded that specific implementation of the presentinvention is merely limited to the descriptions. For those of ordinaryskill in the art, several alternative or evident transformations madewithout departing from the concept of the present invention and havingthe same performance or use shall fall within the protection scope ofthe present invention.

What is claimed is:
 1. A depth map super-resolution processing method,comprising: firstly, performing image acquisition on the same scene in afirst position and a second position, and respectively acquiring a firstoriginal image and a second original image; acquiring a low resolutiondepth map of the first original image; secondly, performing thefollowing processing: 1) dividing the low resolution depth map intomultiple depth image blocks; 2) respectively performing the followingprocessing on the depth image blocks obtained in step 1); 21) performingsuper-resolution processing on a current block with multiplesuper-resolution processing methods, to obtain multiple initial highresolution depth image blocks having the resolution the same as that ofthe first original image; 22) traversing the multiple initial highresolution depth image blocks obtained in step 21), respectivelycombining corresponding image blocks in the first original image whichcorrespond to the current block, and synthesizing multiple image blockscorresponding to the second original image by using an image synthesistechnology according to a relative position relationship between thefirst position and the second position, which are defined as multiplesynthesized image blocks; 23) traversing the multiple synthesized imageblocks obtained in step 22), respectively calculating a matching degreebetween each synthesized image block and a corresponding block in thesecond original image which corresponds to the current block,determining the synthesized image block with the highest matchingdegree, and determining a high resolution depth image blockcorresponding to the synthesized image block with the highest matchingdegree as an ultimate high resolution depth image block of the currentblock; and 3) integrating the high resolution depth image blocks of thedepth image blocks into one image according to positions of the depthimage blocks in the low resolution depth map, to obtain asuper-resolution processing map of the low resolution depth map.
 2. Thedepth map super-resolution processing method according to claim 1,wherein, when the image synthesis technology is adopted, step 22)comprises the following steps: a) projecting the corresponding imageblocks in the first original image into a three-dimensional space byusing a reference camera with a depth-image-based-rendering methodaccording to depth information of the high resolution depth imageblocks; and b) setting the center of a virtual camera by making thecenter of the reference camera correspond to the first position andaccording to the relative position relationship of the second positionrelative to the first position, and imaging a scene of thethree-dimensional space obtained in step a) into a two-dimensional planeby using the virtual camera, so as to obtain a synthesized image block.3. The depth map super-resolution processing method according to claim2, wherein, when projection is conducted in step a), the correspondingimage block is projected into the three-dimensional space according tothe following equation:(X _(w) ,Y _(w) ,Z _(w))^(T) =K ₁ ⁻¹ d ₁ p ₁ wherein the center of thereference camera is the center of the world coordinate system, and adirection observed from the reference camera is a z-axis direction ofthe coordinate system; p1 indicates position information of a pixelpoint p1 in the corresponding image block in the first original image,in a homogeneous form; d1 is depth information of a corresponding pixelpoint p1 in the high resolution depth image block, K1 is a built-inparameter matrix of the reference camera, and (X_(w),Y_(w), Z_(w)) is acoordinate of a point in the three-dimensional space into which thepixel point p1 is projected.
 4. The depth map super-resolutionprocessing method according to claim 2, wherein, when the scene of thethree-dimensional space is imaged to the two-dimensional plane in stepb), the scene of the three-dimensional space is imaged to thetwodimensional plane according to the following equation:d ₂ p ₂ =K ₂ R ₂ P _(w) −K ₂ R ₂ C ₂ wherein the center of the referencecamera is the center of the world coordinate system, and a directionobserved from the reference camera is a z-axis direction of thecoordinate system; C2 is a coordinate of the center of the virtualcamera, R2 is a rotation matrix of the virtual camera, K2 is a built-inparameter matrix of the virtual camera, P_(w) is a coordinate of a pointin the three-dimensional space obtained in step a), p2 and d2 areposition and depth information of the corresponding pixel point in thesynthesized image block obtained through imaging to the two-dimensionalplane; an operation result on the right of the equation is converted toa homogeneous form: m(x, y, 1), p2 is (x, y), and d2 is the coefficientm.
 5. The depth map super-resolution processing method according toclaim 1, wherein, after the high resolution depth image blocks areintegrated into one image, step 3) further comprises smoothing theimage.
 6. The depth map super-resolution processing method according toclaim 1, wherein, when the low resolution depth map is acquired, imageacquisition is performed on the scene in the first position by using adepth camera, so as to acquire the low resolution depth map.